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STATEMENT AS TO RIGHTS TO INVENTIONS MADE UNDER 
FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT 
10 This work was supported by a US Army SBLR contract issued by commander 

of the US Army CECOM, AMSEL-ACCC-A-CF/EW Intelligence Contract No. DAAB07- 
98-C-K004. 

BACKGROUND OF THE INVENTION 

15 Currently, there is great interest in soft-input/soft-output (SISO) algorithms 

due to the outstanding performance of iterative data detection. Like the Viterbi algorithm 
(VA), the SISO algorithm is derived for an embedded system that can be modeled as a finite 
state machine (FSM). However, instead of making hard decisions as does the VA, the SISO 
algorithm generates soft decisions, i.e., reliability measures, for the investigated data. A hard 

20 decision can be readily obtained by thresholding the soft information. The key notion carried 
with the soft information, also its main advantage over a hard decision, is that the soft 
information can be refined by some way (e.g., through iteration) to make a later hard decision 
more reliable. Thus, the SISO algorithm is especially well-suited for iterative data detection. 
Because of this advantage, SISO algorithms have been widely used in various application, 

25 such as decoding of (both serial and parallel) Turbo codes and other codes, near-capacity 

multi-user detection, near-optimal two-dimensional data detection, and the detection of TCM 
signals in fading channels. 

The biggest obstacle facing a practical implementation of the SISO algorithm 
is probably its complexity. The complexity of a SISO algorithm grows exponentially with 

30 the memory length of the FSM, e.g., the number of taps in an inter-symbol interference (ISI) 
channel or the constraint length of a convolutional code. It is not difficult to imagine how the 
complexity of such an algorithm can quickly overwhelm even the most advanced signal 
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processing hardware and software. For a fixed-interval minimum sequence metric (FI-MSM) 
SISO algorithm, its computational complexity is approximately twice that of the VA and the 
number of memory units required is proportional to the product of block length and the 
number of states of the FSM. For instance, when the FI-MSM algorithm is applied to a 10- 
5 tap ISI channel with QPSK signaling, its complexity is dominated by the number of the states 
of the FSM, which is the order of the modulation scheme raised to an exponential equal the 
number of taps minus one (4 9 = 262,144). This relatively simple example of a transmitter and 
channel thus presents an implementation nightmare, as the relevant signal processing system 
is asked to perform SISO computations based on 262,144 possible states, a task that is 
10 practically infeasible today for most applications. 



producing soft outputs based on soft inputs that offers a significant reduction in 
computational and memory complexity from the standard SISO algorithm but is still capable 
of excellent performance. 



According to the invention, in a digital information processing system wherein 
a model of a finite state machine (FSM) receiving a plurality of FSM inputs and producing a 
plurality of FSM outputs is represented by a reduced-state trellis and wherein the FSM inputs 

20 are defined on a base closed set of symbols, a novel method is presented for updating soft 
decision information on the FSM inputs into higher confidence information whereby (1) the 
soft decision information is inputted in a first index set, (2) a forward recursion is processed 
on the input soft decision information based on the reduced-state trellis representation to 
produce forward state metrics, (3) a backward recursion is processed on the input soft 

25 decision information based on the reduced-state trellis representation to produce backward 

state metrics, wherein the backward recursion is independent of the forward recursion and (4) 
the forward state metrics and the backward state metrics are operated on to produce the 
higher confidence information. 

In one embodiment, an a-posteriori probability (APP) RS-SISO algorithm is 

30 used. In another embodiment, a minimum sequence metric (MSM) RS-SISO algorithm is 
used that is computationally simpler than the APP RS-SISO algorithm. 

Results of numerical simulations based on various contexts are also presented 
to illustrate the performance of the RS-SISO algorithm. 



Therefore, there is a need for a method of digital information processing for 
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SUMMARY OF THE INVENTION 
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This invention finds particular application in improvement in the complexity 
and accuracy of data detection in digital signal processing systems. The invention will be 
better understood by reference to the following detailed description in connection with the 
accompanying drawings. 

5 

BRIEF DESCRIPTION OF THE DRAWINGS 
Figure 1 shows a digital communication system employing a trellis coded 
modulation (TCM) encoder; 

Figure 2 shows a section of a trellis describing an FSM which models a 
10 process such as TCM encoding, ISI interference, and others; 

Figure 3 illustrates the construction of the forward and backward transition 
and completion steps at time k in the RS-SISO algorithm; 

Figure 4 shows a RS-SISO module within an iterative detection network; 
Figure 5 A shows an ISI/AWGN channel used test the performance of various 

15 detectors; 

Figure 5B shows a detector representing two hard decision algorithms, the VA 

and DDFSE; 

Figure 5C shows a detector representing the MSM RS-SISO algorithm with 
self-iteration capability; 

20 Figure 6 illustrates the convergence properties of the detector shown in Figure 

5C in different ISI/AWGN channels; 

Figure 7 compares the performance of the MSM RS-SISO algorithm with 
other algorithms in a minimum phase ISI/AWGN channel; 

Figure 8 compares the performance of the MSM RS-SISO algorithm with 
25 other algorithms in non-minimum phase ISI/AWGN channels; 

Figure 9A shows a TCM/ISI/AWGN channel used test the performance of 
various detectors; 

Figure 9B shows the construction of a detector having the VA as both the 
inner and outer detector (VA - VA); 
30 Figure 9C shows the construction of a detector having an inner SISO module 

and an outer VA detector (MSM SISO - VA); 

Figure 9D shows the construction of a detector having an inner RS-SISO 
module and an outer SISO module and employing inner and outer iteration (MSM RS-SISO - 
SISO); and 
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Figure 10 shows the simulation results of the various detectors shown in 
Figures 9B-9D, along with other performance results. 

DESCRIPTION OF SPECIFIC EMBODIMENTS 
5 Figure 1 shows a digital communication system 10 employing a trellis coded 

modulation (TCM) encoder 14. Such a TCM encoder 14 is just one of the many processes 
that can be modeled as a finite state machine (FSM). Other such processes include, but are 
not limited to, encoders based on Turbo codes or other codes, near-capacity multi-user 
detection, near-optimal two-dimensional data detection, fading channels, and inter-symbol 

10 interference (ISI) channels. Referring to Figure 1, the digital communication system 10 
includes a data source 12 providing symbols defined on a particular closed set of symbols. 
For example, if a binary closed set of symbols is used, the symbols are selected from {0, 1}. 
The symbols from the data source 12 are transferred to the TCM encoder 14, which converts 
the symbols into encoded symbols according to the structure of the TCM encoder 14. The 

15 encoded symbols are then sent through a channel 16, which can add noise, such as additive 
white Gaussian noise (AWGN), and distortions to the encoded symbol to produce an 
observed signal. Soft information relating to the observed signal is sent to a TCM decoder 
18. The TCM decoder 18 outputs soft information regarding the symbols which can be 
thresholded to produce hard-decision decoded symbols. Since the operation of the TCM 

20 encoder 14 can be modeled as an FSM, the decoding function of the TCM decoder 18 can be 
based on the reduced-state soft-input/soft-output (RS-SISO) algorithm according to the 
invention as discussed below. 

The digital communication system shown in Figure 1 is a functional block 
diagram providing an illustrative example of basic functions. Different functions may be 

25 added or omitted. Also, application of the innovative methods discussed herein is not limited 
to this specific block diagram. Similarly, other block diagrams discussed below are merely 
illustrative examples and do not limit the applicability of the methods discussed herein. 

Figure 2 shows a section of a trellis describing an FSM which models a 
process such as TCM encoding, ISI interference, and others. Here, the trellis is a state 

30 diagram with explicit denotation of index in time, space, or some other dimension. 

Generally, such a trellis consists of a set of states £ = {Si, £2, . . . , Sy}- The state of the FSM 
at time k is Sk = 5/ e S. The trellis transition t is deterministically driven by a data source that 
outputs sequence {ai, #2, • - . > cik} with symbol a* drawn from the Af-ary closed set of 
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symbols A = {A 0 , A u . . . , A M -\}- Therefore, in an FSM, each transition t is associated with a 
starting state s s (t), an ending state s e (t), an input symbol a(i) and a output symbol x(i). For 
simplicity of presentation, only the FSM whose state is defined as s k = (flk-u &k-L+i> • • * 5 a*— 1) 
is discussed here. However, one skilled in the art would realize that such a restriction is not 
5 necessary to the functionality of the FSM and related operations. An FSM whose state is 
defined as s k = (a> k -u <*k-L+i 9 • • > &k— 1) is said to have memory length L and the number of 
states N = M 1 . Consequently, for a transition t k . . . , a k ) 9 the relevant expressions are: 

s\t k ) = {a k . u . . . , a k .\X (ft) = (a k . L+u ■ • , a k ) 9 a(t k ) = a kj and the output symbol x k = x(t k ). 
The function *(•) can be any mapping, such as a 1-to-l mapping, an n-to-1 mapping, an n-to- 
10 m mapping, or some other mapping. 



simple FSM with N = 4 and M = 2. Thus, the trellis consists of a set of states S = {0, 1 , 2, 3 } . 
The state of the FSM at time k is s k — Si e 5. The trellis transition * is deterministically driven 
by a data source that outputs sequence {a\, a 2 , . . . , a*:} with symbol ^ drawn from the 2-ary 
1 5 closed set of symbols A = {0, 1 } . In other words, the symbols are binary. 



some soft information as its input. The soft input is the sequence of reliability measures of 
the input symbol a k and the output symbol x k . The two main kinds of soft information that 
are widely used are the probability distribution, such as P(a*) and P(x*), and its negative-log 

20 domain counterpart, such as metric M(a k ) and metric M(x k ). Note that the superscript i and o 
to be used with P( ) and M( ) denote "input" and "output", respectively. The resulting 
algorithms are referred to as the a-posteriori probability (APP) algorithm and the minimum 
sequence metric (MSM) algorithm, respectively. It is assumed that the knowledge of FSM 
structure and the a-priori soft information are perfectly known. 

25 The following describes illustrative examples of the reduced-state soft- 

input/soft output (RS-SISO) algorithm. The RS-SISO algorithm achieves complexity 
reduction by a factor of , when compared to the standard SISO algorithm. To implement 
the RS-SISO, the memory of the FSM is truncated and the corresponding trellis is rebuilt. 
The truncated state at time k is defined by v k = {a k -L^a k _^ +lr ..a k _ x } 9 a k where L\<L and L 2 = 

30 L- L\. If L\ = L, then v k = s k , and the RS-SISO algorithm becomes equivalent to the standard 
SISO algorithm. As in the SISO algorithm, the following state sets are defined: 



For clarity of illustration, Figure 2 shows a section of a trellis for a relatively 



Besides the knowledge of the FSM structure, the SISO algorithm also requires 



= {i : V i — > V . is an allowable forward transition} 



(i) 
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B(i) = {j : V. <— Vj is an allowable backward transition} (2) 
C(m) = {j: Vk+\ = Vj, is consistent with ak =A m } (3) 

Figure 3 illustrates the construction of the forward and backward transition 
5 and completion steps at time k in the RS-SISO algorithm. The truncated state at time k is 
v k = {a k _^ a k _^ +l a h _ x ) , and it is illustrated as being concatenated with a forward truncated 

survivor path. The truncated state at time k+1 is v k+x = 0*_^+2, , a k) > ^ * s 

illustrated as being concatenated with a backward truncated survivor path. 

In one embodiment, an a-posteriori probability (APP) RS-SISO algorithm is 
10 used. .Here, forward and backward recursions can be calculated using the following 
recursively-defined quantities: 

<**G) = £ i^ u) a k _ x (i)r f k (ij) (4) 

fik (0 = X j.b(o A+i U)rL 0*, J) (5) 

15 

with initial values as oto (0 = Pr(v<? = V,-) and PkO) = Pr ( v * = v y)- For example, at time k = 0, 
if the initial state of the FSM is V u then ao(/) = 1 , ao(/) = 0, j /. If no knowledge about v 0 is 
available, then ao(0 = . 

Residual state information may be used in the forward and backward 
20 recursions in (4) and (5) by including information on forward and backward truncated 

survivor paths. Associated with each truncated state v* = Vu a truncated survivor path in the 
forward recursion can be obtained as a{ (/) = {a k _ L (/), (/) • • • , a k k _^_ x (/)} , and a backward 
truncated survivor path can be obtained as a* (/) = {a\ (/) , a* +1 (/), • • ■ , a k kJrU _ x 0)} (see Figure 3). 

One reasonable way to obtain these truncated survivor paths is by defining, for a state at time 
25 k, the forward (or backward) survivor state as the state contributing most in the summation of 
(4) or (5). Other reasonable ways of obtaining truncated survivor path are also applicable. 
Based on the truncated survivor paths, the quantities associated with tk{ij) = (v k = V iy v k +j = 
Vj) in both the forward and backward recursion can be defined as (see Figure 3): 



30 



rl 0*> y ) = p' [** = *(*{ (0, t k (i, j ))] p 1 [ a k (i , j)] (6) 
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rt(U)^'[x k+Li =x(t k (i,j), at +I 0))]P'[a A (z,y)] (7) 

where a*(/,y) is the value of the input symbol associated with a forward transition Vi —>Vj. 
When L 2 = 0, y{ (z, j) = y\ (i, j) , yielding the standard APP algorithm. 
5 Note that each element of §{(/) and a*(i) is defined on the M-ary closed set 

of symbols A = {A 0 , A\ 9 . . . , Am-\}* Alternatively, each element of ajf(i) and a*(z) can be 
defined on some set partitioning of the M-ary closed set of symbols A = {Aq, A\ 9 . . . , A M -\}- 
For example, if M= 8 and .4 = {A 0y A u . . . , A 7 }, elements of a{(i) and a£(0 may be defined 

on the set partitioning D = {D 0 , D\ } , where D 0 encompasses {A 0 , A i, . . . , A 3 } and D\ 
10 encompasses {A^ A 5 , . . . , A 7 } . Other types of set partitioning are also possible. 

The complexity of the forward and backward recursion in (4) and (5) is 
primarily determined by the number of states, i.e., M A . Compared to the standard APP- 
SISO algorithm, the complexity of proposed APP RS-SISO is reduced by M 1 ^ times. As 
illustrative examples, two types of soft output can be obtained as: 

15 

= An) = ZyeC( W )^(7)A0') (8) 
^(a k =AJ=cP^a k ^A m )/P i (a k =A m ) , (9) 

where c is a normalization constant. Due to the truncation of states, Y°(x k ) cannot be 
20 yielded directly as in the standard SISO algorithm. However, ¥°(x k ) can still be obtained 
approximately. One simple way is to use the following definition: 

p;u = = nL^ta : '*> (10) 

25 which can work fairly well. It is notable that (10) can be calculated recursively due to the 
temporal relationship between t k and t k +h P e °( ) can be viewed as the (approximate) a- 
posteriori probability normalized to the a-priori probability and is usually referred to as 
"extrinsic" information. If further iteration is necessary, P e °( ) can be used as feedback. 
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Otherwise, the hard decision may be made by the rule: a k = A m if P°(a k = A m ) > 

V°(a k = A j ) for all 0 <j < M- 1 . The completion step in (8) is also illustrated in Figure 3. 

The data detection performance of the RS-SISO algorithm is suboptimal when 
compared to the SISO algorithm, which is more complex in computation and memory. It can 
5 be observed that when the RS-SISO algorithm generates the soft output for the input 
information in time interval [k + 1, k +L 2 ] has not been used. This further degrades 
performance. Self-iteration can be used to improve the performance of the RS-SISO 
algorithm. By feeding its soft-output back to its own soft-input port several times, the RS- 
SISO can fuse the information in the time interval [k + 1, k +L 2 ] into the final soft output for 

10 a k . Self-iteration can thus improve performance significantly. 

The quantities a K and p K are examples of forward state metrics and backward 
state metrics, respectively. Alternatively, the APP RS-SISO algorithm can be based on 
forward transition metrics and backward transition metrics, in addition to forward state 
metrics and backward state metrics. Furthermore, the operations used to obtain .soft output 

15 information need not be limited to summation and multiplication, which is shown in (8) of 
the example discussed above. Such operations may also include summation, multiplication, 
minimum, maximum, minimum*, where minimum* (x, y) = minimum(x, y) - ln(l+e" |x_yl ), 
maximum*, where maximum* (x, y) = maximum(x, y) - ln(l+e" fx " y '), linear weighting, 
exponentiation, combinations of the above, or others. 

20 In another embodiment, a minimum sequence metric (MSM) RS-SISO 

algorithm is used. As in the APP RS-SISO algorithm, residual state information may be used 
in forward and backward recursions by including information on forward and backward 
truncated survivor paths. For example, in the case of an additive white Gaussian noise 
(AWGN), the negative-log domain counterparts to (6) and (7) can be defined as: 

25 

K & J) = M '' [** = *(*{ (0, h (/, y ))] + M' [a k (/, j)] (11) 
4(i,j) = M i [x k+L2 =x(/,(/,7XaL(7))] + M i [a k (ijj\ (12) 

respectively, where the metric M(w) = -ln(P(>v)). 
30 Note that each element of a{(z) and a b k (i) is defined on the M-ary closed set 

of symbols A = {Aq, A\, . . . , A M -\}- Alternatively, each element of a{(z) and §*(*') can be 
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• 
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defined on some set partitioning of the M-ary closed set of symbols A = {Aq 9 A\ 9 . . . , Am-\}- 
For example, if Af = 8 and A = {A 09 A\ 9 . . . 9 A 7 } 9 elements of a{(/) and a* (/) may be defined 

on the set partitioning D = {Do, £>i}, where Z) 0 encompasses {Ao, A\ 9 . . . , A 3 } and A 
encompasses {A 4y A 5 , . . . , A 7 }. Other types of set partitioning are also possible. 

The sequence metric associated with a symbol sequence a£_^ , or the 

equivalent state sequence v£ +I can be defined as: 

A / (vir , )=z;i (i ^«,y) d3) 

A'(vr)=SL,4ay) d") 

and the key quantities for MSM RS-SISO are defined as: 

^0-) = min ^,-^'(0 (15) 
% (0 = min K A 6 (vf +1 ) (16) 

v k+\- v k+l v i 

consequently, the forward and backward recursion and the completion step of MSM RS- 
SISO can be expressed as: 

CO = B^jes-u) V) + d> J)] (17) 

?j k (i) = mm Jem 0') + K+x 0'> 7')] (18) 

M°(a k = AJ = min yeC(m) [S k (j) + fj k (J)] (19) 

M: (a k = A m ) = M°{a k =A m )- M' (a k =A m ) (20) 

m:^=^))=xL m :(«,:o (2d 



25 Since the forward and backward recursion are similar to the recursion in the 

VA, the MSM RS-SISO can obtain the survivor paths used in (10) and (1 1) just as in the VA. 



15 



20 
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Again, M°() is fed back if further iterations are needed, and M°(a k ) is used to make the 
hard decision by the rule: a k = A m if M°(a k = AJ < M°(a k = A y ) for all 0 < j < M -1 . 

Compared to the APP version, the MSM version only involves summation and 
comparison operations. Computationally, it is simpler than the APP version. However, the 
5 way the survivor path is defined in a log- APP algorithm must be specified (i.e., as discussed 
above). Numerical experiments have shown that the MSM SISO algorithm performs almost 
as well as its APP counterpart. Similarly, compared to the standard MSM algorithm, the 
complexity of MSM RS-SISO is also reduced by M h times. 

The quantities S k and jj k are examples of forward state metrics and backward 

10 state metrics, respectively. Alternatively, the MSM RS-SISO algorithm can be based on 
forward transition metrics and backward transition metrics, in addition to forward state 
metrics and backward state metrics. Furthermore, the operations used to obtain soft output 
information need not be limited to minimum and summation, as is shown in (19) of the 
example discussed above. Such operations may also include summation, multiplication, 

15 minimum, maximum, minimum*, where minimum* (x, y) = minimum(x, y) - ln(l+e~ |x ~ yl ), 
maximum*, where maximum* (x, y) = maximum(x, y) - ln(l+e" |x y| ), linear weighting, 
exponentiation, combinations of the above, or others. 

Figure 4 shows a RS-SISO module within an iterative detection network. An 
example of an iterative detection network 40 is shown. Here, a data source 42 is connected to 

20 a TCM encoder 44, which is connected to an interleaver 46, which is connected to an ISI 
channel 48. The output of the ISI channel is sent to the channel 50, where AWGN noise 
and/or distortion may be added. Although pictured separately, the ISI channel 48 maybe a 
part of the channel 50. An iterative detection network 40 consisting of SISO modules and 
soft information exchange rules and schedules receives the output of the channel 50. The 

25 iterative detection network 40 contains an ISI RS-SISO module 52 that has replaced a 

conventional ISI SISO module. It also contains an interleaver/deinterleaver 54 and a TCM 
SISO module 56. Soft information is circulated inside iterative detection network 40 several 
times before output is sent to a thresholder 58, where a hard decisions are made. Usually, the 
maximum number of iteration required to achieve a certain level of performance can be 

30 estimated numerically. For / iterations, the complexity of an iterative detection network 
employing a conventional SISO module, such as the conventional ISI SISO that has been 
replaced here, is proportional to IM L . The RS-SISO algorithm is especially well-suited for 
use in an iterative detection network. By replacing the standard SISO module with an RS- 
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SISO module, such as the ISI RS-SISO module 52, the complexity of the iterative detection 
network employing the RS-SISO algorithm is reduced by a factor of IM 1 ^ . 

The following describes results of numerical simulations performed to test the 
performance of the RS-SISO algorithm. 
5 Figure 5A shows an intersymbol interference, additive white Gaussian noise 

(ISI/AWGN) channel used to test the performance of various detectors. The structure of the 
ISI/AWGN channel is represented as a 12-tap (L = 1 1) ISI block, along with components for 
adding AWGN. As to the taps of the ISI block, which is an FSM, Channel A has equal 
entries and Channel B has taps selected to be (c, 2c* • -,12c) . Both channels are normalized to 

10 have unit power. The transmitter uses a BPSK signaling scheme (i.e., a k = ±^Je^). The 

output of the ISI channel is then corrupted by an AWGN n k with E \n 2 k J = N Q 1 2 . Figure 5B 

shows a detector representing two hard decision algorithms, the VA and the delayed decision- 
feedback sequence estimation (DDFSE) algorithm, that are tested for purposes of 
comparison. Similar to the RS-SISO, truncated state of length L\ is defined in the DDFSE. 
15 Figure 5C shows a detector representing the MSM RS-SISO algorithm with self-iteration 
capability. 

Figure 6 illustrates the convergence properties of the detector shown in Figure 
5C in different ISI/AWGN channels. The performance results shown here and below are 
obtained by numerical simulation. As seen in Figure 6, convergence occurs after 4-5 
20 iterations. 

Figure 7 compares the performance of the MSM RS-SISO algorithm with 
other algorithms in a minimum phase ISI/AWGN channel. An index of the computational 
complexity of the algorithm is defined as the product of the number of transitions M L,+1 , the 
self-iteration number / and the recursion number r. Since the VA is forward-only, it has r = 

25 1. By contrast, the forward/backward SISO has r = 2. In Figures 7, as well as in Figures 8 
and 9, this complexity index is shown for each algorithm used. The performance of the VA 
with 2 11 = 2,048 states is presented as a baseline. First, it is noted that the DDFSE performs 
roughly 3 dB worse than the VA at a bit error rate (BER) of 10" 4 when L\ = 5, i.e., the 
DDFSE uses 2 5 = 32 states. Without self-iteration, the MSM RS-SISO with L\=2 performs 

30 only 0.3 dB better than the DDFSE. However, if self-iteration is used, after only 4 iterations, 
the performance of MSM RS-SISO is improved by 1.9 dB, and is only 0.8 dB away from that 
of the VA. The complexity index indicates that the RS-SISO is 64 times simpler than the VA 
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with a performance degradation of less than 1 dB while it outperforms a DDFSE of similar 
complexity by 2.2 dB. 

Figure 8 compares the performance of the MSM RS-SISO algorithm with 
other algorithms in non-minimum phase ISI/AWGN channels. Due to the bi-directional 
5 recursion of the RS-SISO algorithm (forward and backward recursions), its robustness to 
non-minimum phase channels is expected. By contrast, the DDFSE's non-symmetric 
structure (only a forward recursion is used) degrades its performance greatly when the 
channel is non-minimum phase. Channel B is such a channel, with taps of its ISI block 
selected to be (c, 2c • • • , 12c) . Channel B' is defined as the time-reversed version of Channel 

10 B, that is, (12c,l \c- • • ,c) . The simulation results in Figure 8 clearly show that the DDFSE 
with L\ = 5 virtually fails for Channel B but works well for Channel B\ However, the 
iterative detector based on RS-SISO with the same complexity {L\ = 2 and /= 4) performs 
nearly optimally for both Channels B and B'. 

Figure 9A shows a trellis code modulation, intersymbol interference, additive 

15 white Gaussian noise (TCM/ISI/AWGN) channel used to test the performance of various 

detectors. An 8-state, rate R = 2/3 Ungerboeck 8-PSK TCM code is used in this test. The 8- 
PSK signals from the TCM encoder are fed into a 32><32 block interleaver. The interleaved 
8-PSK signals pass through a 5-tap (L = 4) ISI channel with equal entries (normalized to unit 
power), and the output is corrupted by a white complex circular Gaussian noise rik with 

20 E{\n k \ 2 ) = NJ2. 

Figures 9B-9D show various detectors tested using the signal generated by the 
system shown in Figure 9A. It is too complex to build a Viterbi detector by considering the 
concatenated TCM encoder and ISI channel as a single FSM. Alternatively, an iterative 
detection network may be used. That is, an effective approach is to build a "detector" (hard 

25 or soft) for each subsystem separately, such as an inner and an outer detector, and allow 

information to be processed iteratively and passed between the inner and the outer detector. 
Figure 9B shows the contraction of a detector having the VA as both the inner and outer 
detector (VA - VA). One can use the VA as both the inner and outer detector, but poor 
performance is expected because the TCM code is decoded with hard decisions. Figure 9C 

30 shows the construction of a detector having an inner SISO module and an outer VA detector 
(MSM SISO - VA). Here, the inner VA detector has been replaced by the SISO module, 
which can improve the overall performance since more reliable Euclidean distance is used at 
the outer VA detector. 
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Furthermore, by replacing the outer VA detector by a SISO module and using 
iterative detection, the overall performance can approach that of the optimal performance. 
However, even by treating the subsystems separately, the complexity of the inner detector is 
prohibitive. For example, in the above system, the inner FSM has 8 4 = 4,096 states and 8 5 = 
5 32,768 transitions. 

Figure 9D shows the construction of a detector having an inner RS-SISO 
module and an outer SISO module and employing inner and outer iteration (MSM RS-SISO - 
SISO). Due to the sub-optimality of the RS-SISO algorithm and the concatenated detection 
structure, both the self-iteration of the inner RS-SISO module (also called inner iteration) and 

10 the outer iteration are used to increase the effectiveness of the detector. Therefore, besides 
the number of outer iterations I Q associated with general iterative schemes, there is another 
design parameter for this specific detector: the number of the inner iterations Before 
feeding the soft information on the 8-PSK symbols to the outer SISO, the inner RS-SISO may 
conduct several inner iterations to improve the quality of the soft output. In this case, the 

15 complexity index is redefined as M^IJf . 

Figure 10 shows the simulation results of the various detectors shown in 
Figures 9B-9D, along with other performance results. There is a 5 dB gain in E b I N 0 at a 
BER of 10" 4 by replacing the VA with a MSM-SISO at the inner stage. This can be seen by 
comparing the results for MSM SISO - VA against the result for VA - VA. When replacing 

20 the inner VA by a MSM RS-SISO with L x =2 (7, = 1), a 0.3 dB gain is obtained. This can be 
seen by comparing the results for MSM RS-SISO - VA against the results for VA - VA. In 
other words, without any performance degradation, the detector complexity is reduced by 32 
times by simply using an RS-SISO algorithm, without iterative detection. 

Performance detectors that combine the RS-SISO algorithm with iterative 

25 detection is also illustrated. As a baseline, it is shown that there is a 5 dB gain in E b .l N Q at a 

BER of 10" 4 by replacing the VA with a MSM-SISO at the inner stage. This can be seen by 
comparing the results for MSM SISO - VA against the result for VA - VA. However, the 
complexity of the MSM SISO -VA scheme is prohibitive, as discussed above. By employing 
the RS-SISO algorithm together with iterative detection, complexity can be significantly 
30 reduced. Two such detectors, labeled as MSM RS-RISO - SISO, are tested. Both converge 
in only 4-5 iterations (results are not shown here). One detector uses an MSM RS-SISO with 
L\ = 1 (8 states), and 3 inner iterations (7, = 3). The hard decisions are made after five outer 
iterations (I 0 = 5). Compared to the MSM SISO - VA scheme, the performance degradation 
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is only 1.1 dB while the complexity saving is 34 times. In order to obtain better performance, 
the other detector uses L\ = 2 and h = 3. After four outer iterations (I 0 = 4), a 0.3 dB gain is 
obtained over the MSM SISO - VA scheme while the complexity saving is roughly 5 times. 
Note that in this application, any reduced complexity hard-decision processor (e.g., RSSE, 
5 DDFSE) will perform worse than the VA - VA scheme in Figure 10. 

Thus, simulation results show that the RS-SISO algorithm provides significant 
reductions to complexity in very practical systems. The RS-SISO algorithm can be used in 
conjunction with iterative detection, including self-iteration, to provide additional reductions 
to complexity. Moreover, The results show that favorable tradeoffs between complexity and 
10 performance can be obtained by varying readily adjustable parameters within the algorithm. 

The invention has been explained with reference to specific embodiments. 
Other embodiments will be evident to those of ordinary skill in the art. It is therefore not 
intended that this invention be limited, except as indicated by the appended claims. 
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